More useful man machine interfaces and applications

ABSTRACT

A method for enhancing a well-being of a small child or baby utilizes at least one TV camera positioned to observe one or more points on the child or an object associated with the child. Signals from the TV camera are outputted to a computer, which analyzes the output signals to determine a position or movement of the child or child associated object. The determined position or movement is then compared to preprogrammed criteria in the computer to determine a correlation or importance, and thereby to provide data to the child.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/850,577, filed Mar. 26, 2013 (now U.S. Pat. No. ______), which is acontinuation of U.S. patent Application No. 13/714,748, filed Dec. 14,2012 (now U.S. Pat. No. 8,553,079), which is a continuation of U.S.patent application Ser. No. 12/700,055, filed Feb. 4, 2010, which is acontinuation of U.S. patent application Ser. No. 10/866,191, filed Jun.14, 2004, which is a continuation of U.S. patent application Ser. No.09/433,297, filed Nov. 3, 1999 (now U.S. Pat. No. 6,750,848), whichclaims benefit of U.S. Provisional Application No. 60/107,652, filedNov. 9, 1998. These applications are hereby incorporated by reference.

REFERENCES TO RELATED APPLICATIONS BY THE INVENTORS

U.S. patent application Ser. No. 09/138,339, filed Aug. 21, 1998.

U.S. Provisional Application No. 60/056,639, filed Aug. 22, 1997.

U.S. Provisional Application No. 60/059,561, filed Sep. 19, 1998.

Man Machine Interfaces: Ser. No. 08/290,516, filed Aug. 15, 1994, andnow U.S. Pat. No. 6,008,800.

Touch TV and Other Man Machine Interfaces: Ser. No. 08/496,908, filedJun. 29, 1995, and now U.S. Pat. No. 5,982,352.

Systems for Occupant Position Sensing: Ser. No. 08/968,114, filed Nov.12, 1997, now abandoned, which claims benefit of 60/031,256, filed Nov.12, 1996.

Target holes and corners: U.S. Ser. No. 08/203,603, filed Feb. 28, 1994,and Ser. No. 08/468,358 filed Jun. 6, 1995, now U.S. Pat. No. 5,956,417and U.S. Pat. No. 6,044,183.

Vision Target Based Assembly: U.S. Ser. No. 08/469,429, filed Jun. 6,1995, now abandoned; Ser. No. 08/469,907, filed Jun. 6, 1995, now U.S.Pat. No. 6,301,763; Ser. No. 08/470,325, filed Jun. 6, 1995, nowabandoned; and Ser. No. 08/466,294, filed Jun. 6, 1995, now abandoned.

Picture Taking Method and Apparatus: Provisional Application No.60/133,671, filed May 11, 1998.

Methods and Apparatus for Man Machine Interfaces and Related Activity:Provisional Application No. 60/133,673 filed May 11, 1998.

Camera Based Man-Machine Interfaces: Provisional Patent Application No.60/142,777, filed Jul. 8, 1999.

The copies of the disclosure of the above referenced applications areincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to simple input devices for computers,particularly, but not necessarily, intended for use with 3-D graphicallyintensive activities, and operating by optically sensing object or humanpositions and/or orientations. The invention in many preferredembodiments, uses real time stereo photogrammetry using single ormultiple TV cameras whose output is analyzed and used as input to apersonal computer, typically to gather data concerning the 3D locationof parts of, or objects held by, a person or persons.

This continuation application seeks to provide further detail on usefulembodiments for computing. One embodiment is a keyboard for a laptopcomputer (or stand alone keyboard for any computer) that incorporatesdigital TV cameras to look at points on, typically, the hand or thefinger, or objects held in the hand of the user, which are used to inputdata to the computer. It may also or alternatively, look at the head ofthe user as well.

Both hands or multiple fingers of each hand, or an object in one handand fingers of the other can be simultaneously observed, as canalternate arrangements as desired.

2. Description of Related Art

My referenced co-pending applications incorporated herein by referencediscuss many prior art references in various pertinent fields, whichform a background for this invention.

BRIEF DESCRIPTION OF FIGURES

FIG. 1 illustrates a laptop or other computer keyboard with camerasaccording to the invention located on the keyboard surface to observeobjects such as fingers and hands overhead of the keyboard.

FIG. 2 illustrates another keyboard embodiment using special datums orlight sources such as LEDs.

FIG. 3 illustrates a further finger detection system for laptop or othercomputer input.

FIG. 4 illustrates learning, amusement, monitoring, and diagnosticmethods and devices for the crib, playpen and the like.

FIG. 5 illustrates a puzzle toy for young children having cut out woodcharacters according to the invention.

FIG. 6 illustrates an improved handheld computer embodiment of theinvention, in which the camera or cameras may be used to look atobjects, screens and the like as well as look at the user along thelines of FIG. 1.

FIGS. 7A-B illustrate new methods for internet commerce and otheractivities involving remote operation with 3D virtual objects display.

DESCRIPTION OF THE INVENTION

FIG. 1

A laptop (or other) computer keyboard based embodiment is shown inFIG. 1. In this case, a stereo pair of cameras 100 and 101 located oneach side of the keyboard are used, desirably having cover windows 103and 104 mounted flush with the keyboard surface 102. The cameras arepreferably pointed obliquely inward at angles Φ toward the center of thedesired work volume 170 above the keyboard. In the case of camerasmounted at the rear of the keyboard (toward the display screen), thesecameras are also inclined to point toward the user at an angle as well.

Alternate camera locations may be used such as the positions of cameras105 and 106, on upper corners of screen housing 107 looking down at thetop of the fingers (or hands, or objects in hand or in front of thecameras), or of cameras 108 and 109 shown.

One of the referenced embodiments of the invention is to determine thepointing direction vector 160 of the user's finger (for example pointingat an object displayed on screen 107), or the position and orientationof an object held by the user. Alternatively, finger position data canbe used to determine gestures such as pinch or grip, and other examplesof relative juxtaposition of objects with respect to each other, as hasbeen described in co-pending referenced applications. Positioning of anobject or portions (such as hands or fingers of a doll) is also of use,though more for use with larger keyboards and displays.

In one embodiment, shown in FIG. 2, cameras such as 100/101 are used tosimply look at the tip of a finger 201 (or thumb) of the user, or anobject such as a ring 208 on the finger. Light from below, such asprovided by single central light 122 can be used to illuminate thefinger that typically looks bright under such illumination.

It is also noted that the illumination is directed or concentrated in anarea where the finger is typically located such as in work volume 170.If the light is of sufficient spectral content, the natural flesh toneof the finger can be observed—and recognized by use of the color TVcameras 100/101.

As is typically the case, the region of the overlapping cameras viewingarea is relatively isolated to the overlapping volumetric zone of theirfields 170 shown due to focal lengths of their lenses and the angulationof the camera axes with respect to each other. This restricted overlapzone helps mitigate against unwanted matches in the two images due toinformation generated outside the zone of overlap. Thus there are nosignificant image matches found of other objects in the room, since theonly flesh-toned object in the zone is typically the finger or fingersof the user. Or alternatively, for example, the user's hand or hands.Similarly objects or targets thereon can be distinguished by specialcolors or shapes.

If desired, or required, motion of the fingers can be also used tofurther distinguish their presence vis-a-vis any static background. Iffor example, by subtraction of successive camera frames, the image of aparticular object is determined to have moved it is determined that thisis likely the object of potential interest which can be further analyzeddirectly to determine if is the object of interest.

In case of obscuration of the fingers or objects in the hand, cameras inadditional locations such as those mentioned above, can be used to solvefor position if the view of one or more cameras is obscured.

The use of cameras mounted on both the screen and the keyboard allowsone to deal with obscurations that may occur and certain objects may ormay not be advantageously delineated in one view or the other.

In addition, it may be in many cases desirable to have a datum on thetop of the finger as opposed to the bottom because on the bottom, it canget in the way of certain activities. In this case the sensors arerequired on the screen looking downward or in some other location suchas off the computer entirely and located overhead has been noted inprevious application.

To determine finger location, a front end processor like that describedin the target holes and corners co-pending application referenceincorporated U.S. Ser. Nos. 08/203,603 and 08/468,358 can be used toalso allow the finger shape as well as color to be detected.

Finger gestures comprising a sequence of finger movements can also bedetected by analyzing sequential image sets such as the motion of thefinger, or one finger with respect to another such as in pinchingsomething can be determined. Cameras 100 and 101 have been shown at therear of the keyboard near the screen or at the front. They may mount inthe middle of the keyboard or any other advantageous location.

The cameras can also see one's fingers directly, to allow typing as now,but without the physical keys. One can type in space above the plane ofthe keyboard (or in this case plane of the cameras). This is useful forthose applications where the keyboard of conventional style is too big(e.g., the hand held computer of FIG. 6).

FIG. 2

It is also desirable for fast reliable operation to use retro-reflectivematerials and other materials to augment the contrast of objects used inthe application. For example, a line target such as 200 can be worn on afinger 201, and advantageously can be located if desired between twojoints of the finger as shown. This allows the tip of the finger to beused to type on the keyboard without feeling unusual—the case perhapswith target material on tip of the finger.

The line image detected by the camera can be provided also by a cylindersuch as retroreflective cylinder 208 worn on the finger 201 whicheffectively becomes a line image in the field of view of each camera(assuming each camera is equipped with a sufficiently coaxial lightsource, typically one or more LEDs such as 210 and 211), can be used tosolve easily using the line image pairs with the stereo cameras for thepointing direction of the finger that is often a desired result. Theline, in the stereo pair of images provides the pointing direction ofthe finger, for example pointing at an object displayed on the screen140 of the laptop computer 138.

FIG. 3

It is also possible to have light sources on the finger that can beutilized such as the 2 LED light sources shown in FIG. 3. This can beused with either TV camera type sensors or with PSD type analog imageposition sensors as disclosed in references incorporated.

In particular the ring mounted LED light sources 301 and 302 can bemodulated at different frequencies that can be individually discerned bysensors imaging the sources on to a respective PSD detector.Alternatively, the sources can simply be turned on and off at differenttimes such that the position of each point can be independently foundallowing the pointing direction to be calculated from the LED point datagathered by the stereo pair of PSD based sensors.

The “natural interface keyboard” here described can have cameras orother sensors located at the rear looking obliquely outward toward thefront as well as inward so as to have their working volume overlap inthe middle of the keyboard such as the nearly full volume over thekeyboard area is accommodated.

Clearly larger keyboards can have a larger working volume than one mighthave on a laptop. The pair of sensors used can be augmented with othersensors mounted on the screen housing. It is noted that the linkeddimension afforded for calibration between the sensors located on thescreen and those on the keyboard is provided by the laptop unitaryconstruction.

One can use angle sensing means such as a rotary encoder for the laptopscreen tilt. Alternatively, cameras located on the screen can be used toimage reference points on the keyboard as reference points to achievethis. This allows the calibration of the sensors mounted fixedly withrespect to the screen with respect to the sensors and keyboard spacebelow. It also allows one to use stereo pairs of sensors that are not inthe horizontal direction (such as 101/102) but could for example be acamera sensor such as 100 on the keyboard coupled with one on thescreen, such as 106.

Knowing the pointing angles of the two cameras with respect to oneanother allows one to solve for the 3D location of objects from thematching of the object image positions in the respective camera fields.

As noted previously, it is also of interest to locate a line or cylindertype target on the finger between the first and second joints. Thisallows one to use the fingertip for the keyboard activity but by raisingthe finger up, it can be used as a line target capable of solving forthe pointed direction for example.

Alternatively one can use two point targets on the finger such as eitherretroreflective datums, colored datums such as rings or LED lightsources that can also be used with PSD detectors which has also beennoted in FIG. 2.

When using the cameras located for the purpose of stereo determinationof the position of the fingers from their flesh tone images it is usefulto follow the preprocessing capable of processing data obtained from thecameras in order to look for the finger. This can be done on both colorbasis and on the basis of shape as well as motion.

In this invention, I have shown the use of not only cameras located on ascreen looking downward or outward from the screen, but also camerasthat can be used instead of or in combination with those on the screenplaced essentially on the member on which the keyboard is incorporated.This allows essentially the keyboard to mounted cameras which arepreferably mounted flush with the keyboard surface to be unobtrusive,and yet visually be able to see the users fingers, hands or objects heldby the user and in some cases, the face of the user.

This arrangement is also useful for 3D displays, for example wherespecial synchronized glasses (e.g., the “Crystal Eyes” brand often usedwith Silicon Graphics work stations) are used to alternatively presentright and left images to each eye. In this case the object may appear tobe actually in the workspace 170 above the keyboard, and it may bemanipulated by virtually grasping (pushing, pulling, etc.) it, as hasbeen described in co-pending applications.

FIG. 4: Baby Learning and Monitoring System

A baby's reaction to the mother (or father) and the mother's analysis ofthe baby's reaction is very important. There are many gestures of babiesapparently indicated in child psychology as being quite indicative ofvarious needs, wants, or feelings and emotions, etc. These gestures aretypically made with the baby's hands.

Today this is done and learned entirely by the mother being with thebaby. However with an Electro-optical sensor based computer system, suchas that described in co-pending applications located proximate to oreven in the crib (for example), one can have the child's reactionsrecorded, not just in the sense of a video tape which would be too longand involved for most to use, but also in terms of the actual motionswhich could be computer recorded and analyzed also with the help of themother as to what the baby's responses were. And such motions, combinedwith other audio and visual data can be very important to the baby'shealth, safety, and learning.

Consider for example crib 400 with computer 408 having LCD monitor 410and speaker 411 and camera system (single or stereo) 420 as shown, ableto amuse or inform baby 430, while at the same time recording (bothvisually, aurally, and in movement detected position data concerningparts of his body or objects such as rattles in his hand) his responsesfor any or all of the purposes of diagnosis of his state of being,remote transmission of his state, cues to various programs or images todisplay to him or broadcast to others, or the like.

For one example, baby's motions could be used to signal a response fromthe TV either in the absence of the mother or with the mother watchingon a remote channel. This can even be over the Internet if the mother isat work.

For example, a comforting message could come up on the TV from themother that could be prerecorded (or alternatively could actually belive with TV cameras in the mother's or father's workplace for exampleon a computer used by the parent) to tell the baby something reassuringor comfort the baby or whatever. Indeed the parent can be monitoredusing the invention and indicate something back or even control ateleoperater robotic device to give a small child something to eat ordrink for example. The same applies to a disabled person.

If the father or mother came up on the screen, the baby could wave atit, move its head or “talk” to it but the hand gestures may be the mostimportant.

If the mother knows what the baby is after, she can talk to baby or saysomething, or show something that the baby recognizes such as a doll.After a while, looking at this live one can then move to talking to thebaby from some prerecorded data.

What other things might we suppose? The baby for example knows to putsits hand on the mother's cheek to cause the mother to turn to it. Thebaby also learns some other reflexes when it is very young that itforgets when it gets older. Many of these reflexes are hand movements,and are important in communicating with the remote TV based motherrepresentation, whether real via telepresense or from CD Rom or DVD disk(or other media, including information transmitted to the computer fromafar) and for the learning of the baby's actions.

Certainly just from the making the baby feel good point-of-view, itwould seem like certain motherly (or fatherly, etc.) responses tocertain baby actions in the form of words and images would be useful.This stops short of physical holding of the baby which is often needed,but could act as a stop gap to allow the parents to get another hour'ssleep for example.

As far as the baby touching things, I've discussed in other applicationsmethods for realistic touch combined with images. This leads to a newform of touching crib mobiles that could contain video imaged and or beimaged themselves—plus if desired—touched in ways that would be farbeyond any response that you could get from a normal mobile.

For example, let us say there is a targeted (or otherwise TV observable)mobile 450 in the crib above the baby. Baby reaches up and touches apiece of the mobile which is sensed by the TV camera system (either fromthe baby's hand position, the mobile movement, or both, and a certainsound is called up by the computer, a musical note for example. Anotherpiece of the mobile and another musical note. The mobile becomes amusical instrument for the baby that could play either notes or chordsor complete passages, or any other desired programmed function.

The baby can also signal things. The baby can signal using agitatedmovements would often mean that it's unhappy. This could be interpretedusing learned movement signatures and artificial intelligence as neededby the computer to call for mother even if the baby wasn't crying. Ifthe baby cries, that can be picked up by microphone 440, recognizedusing a voice recognition system along the lines of that used in IBM ViaVoice commercial product for example. And even the degree of crying canbe analyzed to determine appropriate action.

The computer could also be used to transmit information of this sort viathe internet email to the mother who could even be at work. And untilhelp arrives in the form of mother intervention or whatever, thecomputer could access a program that could display on a screen for thebaby things that the baby likes and could try to soothe the baby througheither images of familiar things, music or whatever. This could beuseful at night when parents need sleep, and anything that would makethe baby feel more comfortable would help the parents.

It could also be used to allow the baby to input to the device. Forexample, if the baby was hungry, a picture of the bottle could bebrought up on the screen. The baby then could yell for the bottle. Or ifthe baby needed his diaper changed, perhaps something reminiscent ofthat. If the baby reacts to such suggestions of his problem, this givesa lot more intelligence as to why he is crying and while mothers cangenerally tell right away, not everyone else can. In other words, thisis pretty neat for babysitters and other members of the household sothey can act more intelligently on the signals the baby is providing.

Besides in the crib, the system as described can be used in conjunctionwith a playpen, hi-chair or other place of baby activity.

As the child gets older, the invention can further be used also withmore advanced activity with toys, and to take data from toy positions aswell. For example, blocks, dolls, little cars, and moving toys even suchas trikes, scooters, driveable toy cars and bikes with training wheels.

The following figure illustrates the ability of the invention to learn,and thus to assist in the creation of toys and other things.

FIG. 5: Learning Puzzle Roy

Disclosed in FIG. 5 is a puzzle toy 500 where woodcut animals such asbear 505 and lion 510 are pulled out with handle such as 511. The childcan show the animal to the camera and a computer 530 with TV camera (orcameras) 535 can recognize the shape as the animal, and provide asuitable image and sounds on screen 540.

Alternatively, and more simply, a target, or targets on the back of theanimal can be used such as triangle 550 on the back of lion 511. Ineither case the camera can solve for the 3D, and even 5 or 6D positionand orientation of the animal object, and cause it to move accordinglyon the screen as the child maneuvers it. The child can hold two animals,one in each hand and they can each be detected, even with a singlecamera, and be programmed in software to interact as the child wishes(or as he learns the program).

This is clearly for very young children of two or three years of age.The toys have to be large so they can't be swallowed.

With the invention in this manner, one can make a toy of virtuallyanything, for example a block. Just hold this block up, teach thecomputer/camera system the object and play using any program you mightwant to represent it and its actions. To make this block known to thesystem, the shape of the block, the color of the block or some code onthe block can be determined. Any of those items could tell the camerawhich block it was, and most could give position and orientation ifknown.

At that point, an image is called up from the computer representing thatparticular animal or whatever else the block is supposed to represent.Of course this can be changed in the computer to be a variety of thingsif this is something that is acceptable to the child. It could certainlybe changed in size such as a small lion could grow into a large lion.The child could probably absorb that more than a lion changing into agiraffe for example since the block wouldn't correspond to that. Thechild can program or teach the system any of his blocks to be the animalhe wants and that might be fun.

For example, he or the child's parent could program a square to be agiraffe where as a triangle would be a lion. Maybe this could be aninteresting way to get the child to learn his geometric shapes!

Now the basic block held up in front of the camera system could belooked at just for what it is. As the child may move the thing toward oraway from the camera system, one may get a rough sense of depth from thechange in shape of the object. However this is not so easy as the objectchanges in shape due to any sort of rotations.

Particularly interesting then is to also sense the rotations if theobject so that the animal can actually move realistically in 3Dimensions on the screen. And perhaps having the de-tuning of the shapeof the movement so that the child's relatively jerky movements would notappear jerky on the screen or would not look so accentuated. Converselyof course, you can go the other way and accentuate the motions.

This can, for example, be done with a line target around the edge of theobject is often useful for providing position or orientation informationto the TV camera based analysis software, and in making the objecteasier to see in reflective illumination.

Aid to Speech Recognition

The previous co-pending application entitled “Useful man machineinterfaces and applications” referenced above, discussed the use ofpersons movements or positions to aid in recognizing the voice spoken bythe person.

In one instance, this can be achieved by simply using ones hand toindicate to the camera system of the computer that the voice recognitionshould start (or stop, or any other function, such as a paragraph orsentence end, etc.).

Another example is to use the camera system of the invention todetermine the location of the persons head (or other part), from whichone can instruct a computer to preferentially evaluate the sound fieldin phase and amplitude of two or more spaced microphones to listen fromthat location—thus aiding the pickup of speech—which often times is notable to be heard well enough for computer based automatic speechrecognition to occur.

Digital Interactive TV

As you watch TV, data can be taken from the camera system of theinvention and transmitted back to the source of programming. This couldinclude voting on a given proposition by raising your hand for example,with your hand indication transmitted. Or you could hold up 3 fingers,and the count of fingers transmitted. Or in a more extreme case, yourposition, or the position of an object or portion thereof could betransmitted—for example you could buy a coded object—whose code would betransmitted to indicate that you personally (having been pre-registered)had transmitted a certain packet of data.

-   -   If the programming source can transmit individually to you (not        possible today, but forecast for the future), then much more is        possible. The actual image and voice can respond using the        invention to positions and orientations of persons or objects in        the room—just as in the case of prerecorded data—or one to one        internet connections. This allows group activity as well.

In the extreme case, full video is transmitted in both directions andtotal interaction of users and programming sources and each otherbecomes possible.

An interim possibility using the invention is to have a programbroadcast to many, which shifts to prerecorded DVD disc or the likedriving a local image, say when your hand input causes a signal to beactivated.

Handwriting Authentication

A referenced co-pending application illustrated the use of the inventionto track the position of a pencil in three dimensional space such thatthe point at which the user intends the writing point to be at, can beidentified and therefore used to input information, such as the intendedscript.

As herein disclosed, this part of the invention can also be used for thepurpose of determining whether or not a given person's handwriting orsignature is correct.

For example, consider authentication of an Internet commercialtransaction. In this case, the user simply writes his name or addressand the invention is used to look at the movements of his writinginstrument and determine from that whether or not the signature isauthentic. (A movement of one or more of his body parts might also oralternatively be employed). For example a series of frames of datumlocation on his pen can be taken, to determine one or more positions onit as a function of time, even to include calculating of its pointingdirection, from a determined knowledge in three axes of two points alongthe line of the pen axis. In this case a particular pointing vectorsequence “signature” would be learned for this person, and compared tolater signatures.

What is anticipated here is that in order to add what you might call theconfirming degree of authenticity to the signature, it may not benecessary to track the signature completely. Rather one might onlydetermine that certain aspects of the movement of the pencil are theauthentic ones. One could have people write using any kind of movement,not just their signature having their name. The fact is that people aremostly used to writing their name and it would be assumed that thatwould be it. However, it could well be that the computer asks the userto write something else that they would then write and that particularthing would be stored in the memory.

Optionally, one's voice could be recognized in conjunction with themotion signature to add further confirmation.

This type of ability for the computer system at the other end of theInternet to query a writer to write a specific thing in a random fashionadds a degree of cryptographic capacity to the invention. In otherwords, if I can store the movements in my hand to write differentthings, then clearly this has some value.

The important thing though is that some sort of representation of themovements of the pencil or other instrument can be detected using theinvention and transmitted.

FIG. 6: Hand Held Computer

FIG. 6 illustrates an improved handheld computer embodiment of theinvention. For example, FIG. 8 of the provisional application referencedabove entitled “camera based man machine interfaces and applications”illustrates a basic hand held device and which is a phone, or a computeror a combination thereof, or alternatively to being hand held, can be awearable computer for example on one's wrist.

In this embodiment, we further disclose the use of this device as acomputer, with a major improvement being the incorporation of a cameraof the device optionally in a position to look at the user, or an objectheld by the user—along the lines of FIG. 1 of the instant disclosure forexample.

Consider hand held computer 901 of FIG. 6, incorporating a camera 902which can optionally be rotated about axis 905 so as to look at the useror a portion thereof such as finger 906, or at objects at which it ispointed. Optionally, and often desirably, a stereo pair of cameras tofurther include camera 910 can also be used. It too may rotate, asdesired. Alternatively fixed cameras can be used as in FIG. 1, and FIG.8 of the referenced co-pending application, when physical rotation isnot desired, for ruggedness, ease of use, or other reasons (noting thatfixed cameras have fixed fields of view, which limit versatility in somecases).

When aimed at the user, as shown, it can be used, for example, to viewand obtain images of:

Ones self-facial expression etc., also for image reasons, id etc.,combined effect.

Ones fingers (any or all), one finger to other and the like. This inturn allows conversing with the computer in a form of sign languagewhich can replace the keyboard of a conventional computer.

One or more objects in one's hand. Includes a pencil or pen, and thuscan be used rather than having a special touch screen and pencil if thepencil itself is tracked as disclosed in the above figure. It alsoallows small children to use the device, and those who cannot hold anordinary stylus.

One's Gestures.

The camera 902 (and 910 if used, and if desired), can also be optionallyrotated and used to viewpoints in space ahead of the device, as shown indotted lines 902 a. In this position for example it can be used for thepurposes described in the previous application. It can also be used toobserve or point at (using optional laser pointer 930) points such as935 on a wall, or a mounted LCD or projection display such as 940 on awall or elsewhere such as on the back of an airline seat.

With this feature of the invention, there is no requirement to carry acomputer display with you as with a infrared connection (not shown) suchas known in the art one can also transmit all normal control informationto the display control computer 951. As displays become ubiquitous, thismakes increasing sense—otherwise the displays get bigger the computerssmaller trend doesn't make sense if they need to be dragged aroundtogether. As one walks into a room, one uses the display or displays inthat room (which might themselves be interconnected).

The camera unit 902 can sense the location of the display in spacerelative to the handheld computer, using for example the four points955-958 on the corners of the display as references. This allows thehandheld device to become an accurate pointer for objects displayed onthe screen, including control icons. And it allows the objects on thescreen to be sensed directly by the camera—if one does not have thecapability to spatially synchronize and coordinate the display driverwith the handheld computer.

The camera can also be used to see gestures of others, as well as theuser, and to acquire raw video images of objects in its field.

A reverse situation also exists where the cameras can be on the wallmounted display, such as cameras 980 and 981 can be used to look at thehandheld computer module 901 and determine its position and orientationrelative to the display.

Note that a camera such as 902, looking at you the user, if attached tohand held unit, always has reference frame of that unit. If one workswith a screen on a wall, one can aim the handheld unit with camera atit, and determine its reference frame to the handheld unit. Also canhave two cameras operating together, one looking at wall thing, other atyou (as 902 and 902 a) in this manner, one can dynamically compare refframes of the display to the human input means in determining displayparameters. This can be done in real time, and if so one can actuallywave the handheld unit around while still imputing accurate data to thedisplay using ones fingers, objects or whatever.

Use of a laser pointer such as 930 incorporated into the handheld unithas also been disclosed in the referenced co-pending applications. Forexample, a camera on the hand held computer unit such as 902 viewing indirection 902 a would look at laser spot such as 990 (which might ormight not have come from the computers own laser pointer 930) on thewall display say, and recognized by color and size/shape reference toedge of screen, and to projected spots on screen.

FIGS. 7A-B: Internet and Other Remote Applications

FIG. 7A illustrates new methods for internet commerce and otheractivities involving remote operation with 3D virtual objects displayedon a screen. This application also illustrates the ability of theinvention to prevent computer vision eye strain.

Let us first consider the operation of the invention over the internetas it exists today in highly bandwidth limited form dependent onordinary phone lines for the most part. In this case it is highlydesirable to transmit just the locations or pointing vectors of portions(typically determined by stereo photo-grammetry of the invention) ofhuman users or objects associated therewith to a remote location, toallow the remote computer 10 to modify the image or sound transmittedback to the user.

Another issue is the internet time delay, which can exist in varyingdegrees, and is more noticeable, the higher resolution of the imagerytransmitted. In this case, a preferred arrangement is to have real timetransmission of minimal position and vector data (using no morebandwidth than voice), and to transmit back to the user, quasistationary images at good resolution. Transmission of low resolutionnear real time images common in internet telephony today, does notconvey the natural feeling desired for many commercial applications tonow be discussed. As bandwidth becomes more plentiful these restrictionsare eased.

Let us consider the problem posed of getting information from theinternet of today. A user 1000 can go to a virtual library displayed onscreen 1001 controlled by computer 1002 where one sees a group 1010 ofbooks on stacks. Using the invention as described herein andincorporated referenced applications to determine my hand and fingerlocations, I the user, can point at a book such as 1014 in a computersensed manner, or even reach out and “grab” a book, such as 1020 (dottedlines) apparently generated in 3D in front of me.

My pointing, or my reach and grab is in real time, and the vector (suchas the pointing direction of ones finger at the book on the screen, orthe position and orientation closing vectors of one's forefinger andthumb to grab the 3D image 1020 of the book) indicating the book inquestion created is transmitted back by internet means to the remotecomputer 1030 which determines that I have grabbed the book entitled Warand Peace from the virtual shelf. A picture of the book coming off theshelf is then generated using fast 3D graphical imagery such as theMerlin VR package available today from Digital Immersion company ofSudbury, Ontario. This picture (and the original picture of the books onthe shelves) can be retransmitted over the internet at low resolution(but sufficient speed) to give a feeling of immediacy to the user. Oralternatively, the imagery can be generated locally at higher resolutionusing the software package resident in the local computer 1002 whichreceives key commands from the distant computer 1030.

After the book has been “received” by the user, it then can be openedautomatically to the cover page for example under control of thecomputer, or the users 10 hands can pretend to open it, and the sensedhands instruct the remote (or local, depending on version) computer todo so. A surrogate book such as 1040 can also be used to give the user atactile feel of a book, even though the real book in questions pageswill be viewed on the display screen 1001. One difference to this couldbe if the screen 1001 depicting the books were life size, like realstacks. Then one might wish to go over to a surrogate book incorporatinga separate display screen—just as one would in a real library, go to areading table after removing a book from a stack.

Net Grocery stores have already appeared, and similar applicationsconcern picking groceries off of the shelf of a virtual supermarket, andfilling ones shopping cart. For that matter, any store where it isdesired to show the merchandise in the very manner people are accustomedto seeing it, namely on shelves or racks, generally as one walks down anaisle, or fumbles through a rack of clothes for example. In each case,the invention, which also can optionally use voice input, as if to talkto a clothing sales person, can be used to monitor the person'spositions and gestures.

The invention in this mode can also be used to allow one to peruse muchlarger objects. For example, to buy a car (or walk through a house, say)over the internet, one can lift the hood, look inside, etc., all byusing the invention to monitor the 3D position of your head or hands andmove the image of the car presented accordingly. If the image ispresented substantially life-size, then one can be monitored as onephysically walks around the car in one's room say, with the imagechanging accordingly. In other words just as today.

Note that while the image can be apparently life-size using virtualreality glasses, the natural movements one is accustomed to in buying acar are not present. This invention makes such a natural situationpossible (though it can also be used with such glasses as well).

It is noted that the invention also comprehends adding a force basedfunction to a feedback to your hands, such that it feels like you liftedthe hood, or grabbed the book, say. For this purpose holding a surrogateobject as described in co-pending applications could be useful, in thiscase providing force feedback to the object.

If one looks at internet commerce today, some big applications haveturned out 10 to be clothes and books. Clothes are by far the largestexpenditure item, and let's look closer at this.

Consider too a virtual mannequin, which can also have measurements of aremote shopper. For example, consider diagram 78, where a woman'smeasurements are inputted by known means such as a keyboard 1050 overthe internet to a CAD program in computer 1055, which creates on displayscreen 1056 a 3D representation of a mannequin 1059 having the woman'sshape in the home computer 1060. As she selects a dress 1065 to try on,the dress which let's say comes in 10 sizes, 5 to 15, is virtually“tried on” the virtual mannequin and the woman 1070 looks at the screen1056 and determines the fit of a standard size 12 dress. She can rapidlyselect larger or smaller sizes and decide which she thinks looks and/orfits better.

Optionally, she can signal to the computer to rotate the image in anydirection, and can look at it from different angles up or down as well,simply doing a rotation in the computer. This signaling can beconventional using for example a mouse, or can be using TV based sensingaspects of the invention such as employing camera 1070 also as shown inFIG. 1 for example. In another such case, she can reach out with herfinger 1075 for example, and push or pull in a virtual manner thematerial, using the camera to sense the direction of her finger. Or shecan touch herself at the points where the material should be taken up orlet out, with the camera system sensing the locations of touch(typically requiring at least a stereo pair of cameras or otherelectro-optical system capable of determining where her fingertip is in3D space. Note that a surrogate for the tried on dress in this case,could be the dress she has on, which is touched in the location desiredon the displayed dress.

The standard size dress can then be altered and shipped to her, or therequisite modifications can be made in the CAD program, and a specialdress cut out and sewed which would fit better.

A person can also use her hands via the TV cameras of the invention todetermine hand location relative to the display to take clothes off avirtual manikin which could have a representation of any person real orimaginary. Alternatively she can remotely reach out using the inventionto a virtual rack of clothes such as 1090, take an object off the rack,and put it on the manikin. This is particularly natural in nearlife-size representation, just like being in a store or other venue.This ability of the invention to bring real life experience to computershopping and other activity that is a major advantage.

The user can also feel the texture of the cloth if suitable hapticdevices are 15 available to the user, which can be activated remotely bythe virtual clothing program, or other type of program.

Modifications of the invention herein disclosed will occur to personsskilled in the art, and all such modifications are deemed to be withinthe scope of the invention as defined by the appended claims.

1.-20. (canceled)
 21. A laptop computer comprising: a computer housingincluding a first portion and a second portion, the first portionincluding a display and the second portion including a keyboard, thefirst portion being movable relative to the second portion to an openposition; an electro-optical sensor supported by the first portion andoriented toward a work volume above the second portion when the firstportion is in the open position, the electro-optical sensor providing anoutput; and a processing unit coupled to the electro-optical sensoroutput and adapted to determine a three-dimensional location of aportion of a person in the work volume above the second portion.
 22. Thelaptop computer of claim 21 wherein the processing unit is adapted todetermine a gesture.
 23. The laptop computer of claim 21 wherein theprocessing unit is adapted to determine a pointing direction of a fingerof the person.
 24. The laptop computer of claim 21 wherein theprocessing unit is adapted to determine a relative location of twoportions of said person.
 25. The laptop computer of claim 22 wherein theprocessing unit is adapted to compare successive outputs of theelectro-optical sensor to determine the gesture.
 26. The laptop computerof claim 22 wherein the gesture includes at least one of a push gesture,a pull gesture, a grasp gesture and a pinch gesture.
 27. The laptopcomputer of claim 21 further including a light source adapted toilluminate said portion within the work volume.
 28. The laptop computerof claim 27 wherein the light source includes at least one lightemitting diode.
 29. The laptop computer of claim 21 wherein theelectro-optical sensor includes first and second cameras.
 30. The laptopcomputer of claim 24 wherein the first portion defines first and secondapertures for the first and second cameras.
 31. A computer apparatuscomprising: a laptop housing; a display incorporated into the laptophousing; an electro-optical sensor incorporated into the laptop housingproximate the display, the electro-optical sensor providing an output;and a processing unit adapted to determine a three-dimensional locationof a portion of a person based on the output of the electro-opticalsensor, wherein the three-dimensional location of the portion of aperson is used to input data into the computer apparatus.
 32. Thecomputer apparatus of claim 31 wherein the processing unit is adapted todetermine a gesture.
 33. The computer apparatus of claim 31 wherein theprocessing unit is adapted to determine a pointing direction of theperson.
 34. The computer apparatus of claim 31 wherein the processingunit is adapted to determine a relative location of at least twoportions of the person.
 35. The computer apparatus of claim 32 whereinthe gesture includes at least one of a push gesture, a pull gesture, agrasp gesture and a pinch gesture.
 36. The computer apparatus of claim32 wherein the processing unit is adapted to compare successive outputsof the electro-optical sensor to determine the gesture.
 37. The computerapparatus of claim 31 wherein the electro-optical sensor includes firstand second cameras.
 38. The computer apparatus of claim 31 wherein thedisplay is a three-dimensional display.
 39. The computer apparatus ofclaim 31 further including a light source incorporated into the laptophousing.
 40. The computer apparatus of claim 39 wherein the light sourceincludes at least one light emitting diode.